Image device that accelerates reconstruction process

ABSTRACT

An imaging device includes: a lens optical system, for focusing light from a subject; a single chip color imaging element equipped with a Bayer pattern color filter, for imaging an image of the subject focused by the lens optical system; and an image processing section, for performing a filtering process in which data output by the imaging element is passed through an image reconstructing filter having properties inverse blur properties of the optical system, and then performing a synchronization process. The image processing section collects data excluding zero elements for each of R, G, and B channels, to generate reduced data arrays in which the amount of data is ¼ for the R and B channels, and ½ for the G channel, and administers the filtering process using the image reconstruction filter onto data of the reduced data array for each of the R, G, and B channels.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is related to an imaging device. Moreparticularly, the present invention is related to an imaging device thatenables obtainment of color images in a focused state regardless of thedistance to a subject.

2. Description of the Related Art

An imaging device has been proposed, in which the spatial frequencyproperties of a lens optical system are stabilized by inserting a phaseplate, and image reconstruction processes, that is, processes in whichimage signals are passed through reconstructing filters havingproperties inverse the blur properties of the imaging lens opticalsystems, are administered to enable obtainment of images in a focusedstate regardless of the distances to subjects. U.S. Patent ApplicationPublication No. 20090147111 and Japanese Unexamined Patent PublicationNo. 2009-089082 describe examples of such an imaging device.

It is often the case that the aforementioned type of imaging deviceemploys an imaging element constituted by a CCD or the like, similar togeneral imaging devices. In this case, a single chip color imagingelement, in which a color filter constituted by R (red), G (green), andB (blue) filters for each pixel arranged in a two dimensional matrix isprovided on a photoelectric converting section, is often employed, toperform imaging of color images.

U.S. Patent Application Publication No. 20090147111 and JapaneseUnexamined Patent Publication No. 2009-089082 propose methods foradministering the aforementioned image reconstruction process when usingsuch a single chip color image element in detail.

First, U.S. Patent Application Publication No. 20090147111 proposes toperform the image reconstruction process separately for each of R, G,and B channels. Note that in this proposed method, a Bayer pattern colorfilter is employed, and therefore, with respect to the G channel, theimage reconstruction process is performed on a combined G channel, whichis a Gr and Gb channel on which sensitivity correction has beenadministered and combined. In this reconstruction process, areconstruction filter is employed that performs convolution such that ¾of the elements of the R and B channels become zero and 2/4 of theelements of the G channel become zero in a zigzag arrangement.

Meanwhile, Japanese Unexamined Patent Publication No. 2009-089082proposes to generate a reconstruction filter for data of each of R, Gr,Gb, and B channels, and employing the generated reconstruction filtersto independently perform reconstruction processes for each channel, inthe case that such a color imaging element is employed.

In the method described in U.S. Patent Application Publication No.20090147111, calculating processes (branched calculations) are performedfor elements of which the value of the reconstruction filter is zero,and therefore it is recognized that there is a problem of highcalculation costs. In addition, with respect to memory capacity forstoring data of the reconstruction filter, four times the memorycapacity is necessary for elements having values other than zero for theR and B channels, and twice the memory capacity is necessary forelements having values other than zero for the G channel. For thesereasons, this method is not economical.

Meanwhile, in the method described in Japanese Unexamined PatentPublication No. 2009-089082, convolution calculating processes, whichhave extremely high calculation costs, are administered for all pixels.Therefore, a problem is recognized that the calculation cost is high inthis case as well. In addition, it is known that the G channel greatlyinfluences the perceived resolution of ultimately obtained images.However, because the method described in this patent document performsimage reconstruction processes on the Gr channel and Gb channelseparately, there is a problem that high frequency components are likelyto be lost in reconstructed images.

SUMMARY OF THE INVENTION

The present invention has been developed in view of the foregoingcircumstances. It is an object of the present invention to sufficientlyreduce calculation costs (calculation time/amount of memory) whenobtaining color images employing a single chip color imaging element inan imaging device that enables obtainment of images in a focused stateregardless of distances to subjects. It is a further object of thepresent invention to prevent loss of high frequency components inreconstructed images.

An imaging device of the present invention comprises:

a lens optical system, for focusing light from a subject;

color imaging means equipped with a Bayer pattern color filter, forimaging an image of the subject focused by the lens optical system; and

an image processing section, for performing a filtering process in whichdata output by the imaging means is passed through an imagereconstructing filter having properties inverse the blur properties ofthe lens optical system, then performing a synchronization process;

the image processing section collecting data excluding zero elements foreach of R, G, and B channels, to generate reduced data arrays in whichthe amount of data is ¼ for the R and B channels, and the amount of datais ½ for the G channel, and administering the filtering process usingthe image reconstruction filter onto data of the reduced data array foreach of the R, G, and B channels.

Note that it is desirable for the image processing section to beconfigured to:

collect data excluding zero elements, and rotates the data array 45degrees to generate the reduced data array for the G channel;

administer the filtering process employing an image reconstructionfilter, of which data have similarly been rotated 45 degrees, onto datathat constitutes the reduced data array for the G channel; and

rotate the data obtained by the filtering process −45 degrees, to returnthe orientation of the data array.

Alternatively, the image processing section may be configured to collectdata excluding zero elements in one of the horizontal direction and thevertical direction for the G channel.

It is desirable for correction gain, for correcting differences insensitivities among Gr cells and Gb cells, to be overlapped onto eachelement of the image reconstruction filter which is employed withrespect to data of the G channel.

As described above, the image processing section of the imaging deviceof the present invention collects data for each of the R, G, and Bchannels while excluding zero elements. Thereby, reduced data arrays, inwhich the amounts of data for the R and B channels is ¼, and the amountof data for the G channel is ½, are generated. The filtering processusing the image reconstruction filter is performed onto data thatconstitutes the reduced data arrays for each of the R, G, and Bchannels. Therefore, zero elements of the image reconstruction filterare eliminated from the calculation processes, which reduces thecalculation costs. Further, the configurations of calculating programsand circuits can be simplified.

In addition, the imaging device of the present invention performs theimage reconstruction process onto the G channel, which is a combinationof the Gr channel and the Gb channel. Therefore, loss of high frequencycomponents which occurs when the Gr channel and the Gb channel areprocessed separately can be prevented, and a more highly detailedultimate image can be obtained.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram that illustrates the construction ofan imaging device according to an embodiment of the present invention.

FIG. 2 is a flow chart that illustrates the steps of image processesperformed by the imaging device of FIG. 1.

FIG. 3 is a schematic diagram that illustrates a Bayer pattern imageobtained by the imaging device of FIG. 1.

FIG. 4 is a diagram for explaining reduced data arrays for each of R, G,and B channels within the imaging device of FIG. 1.

FIG. 5 is a diagram for explaining the reduced data arrays and imagereconstruction filters.

FIG. 6 is a diagram for explaining an image which is ultimately obtainedby the imaging device of FIG. 1.

FIG. 7 is a diagram for explaining the reduced data arrays and imagereconstruction filters of a second embodiment of the present invention.

FIG. 8 is a diagram for explaining image reconstruction filters and dataarrangement following image reconstruction processes in the secondembodiment.

FIG. 9 is a diagram for explaining image reconstruction filters and dataarrangement following image reconstruction processes in a thirdembodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention will be described indetail with reference to the attached drawings.

FIG. 1 is a schematic block diagram that illustrates the basicconstruction of a color imaging device according to a first embodimentof the present invention. As illustrated in FIG. 1, the imaging deviceis equipped with: a lens optical system 12 constituted by a lens 10 anda phase plate 11, for example, for focusing light from a subject; acolor imaging element 13 constituted by a CCD, for example, for imagingan image of the subject focused by the lens optical system 12; an A/Dconverter 14 for digitizing analog output from the color imaging element13; an image reconstruction processing section 15, for administering animage reconstruction process to reduce image blur onto image data outputfrom the A/D converter 14; deconvolution filters 16 for performing theimage reconstruction process; a memory 17 connected to the imagereconstruction processing section 15 as a workspace; an interpolationprocessing section 18, for administering an interpolation process withrespect to data output from the image reconstruction processing section15; a memory 19 connected to the interpolation processing section 18 asa workspace; and an image output section 20, for outputting images basedon R, G, and B image data output from the interpolation processingsection 18.

Note that in the present embodiment, the aforementioned elements 15through 19 are constituted by known computer systems. The elements 15through 19 constitute an image processing section of the presentinvention. The image output section 20 may be a recording device thatrecords images onto recording media, or display means that displaysimages employing a CRT, a liquid crystal display panel, etc.

Hereinafter, the processes performed by the image reconstructionprocessing section 15 and the interpolation processing section 18 willbe described with reference to FIG. 2, which illustrates the steps ofthe processes. When the process starts, the image reconstructionprocessing section 15 first separates the digital color image dataoutput from the A/D converter into three images, for each of an R, a G,and a B channel (step S1 of FIG. 2). This is illustrated in columns 1and 2 (counted from the left edge) of FIG. 3.

Here, a single chip color imaging element having a color filter in theBayer pattern is employed as the color imaging element 13. The imagesborne by the image data, which are the analog signals output from thecolor imaging element 13 and digitized by the A/D converter 14, areBayer pattern images, in which R images, G images (more specifically, Grimages which are alternately arranged with R images and constitute asingle line, and Gb images which are alternately arranged with B imagesand constitute a single line), and B images are arranged, as illustratedin FIG. 3.

In the present embodiment, the number of data in the horizontaldirection of the Bayer pattern image is designated as W, and the numberof data in the vertical direction is designated as H, as illustrated inFIG. 3. In addition, a point spread is indicated by the circles A drawnby a broken line in FIG. 3. The diameter of the point spread isdesignated as φ. Note that in the example of FIG. 3, φ is equivalent tothe length of five cells.

Next, the image reconstruction processing section 15 closes the gapswithin each of the R, G, and B images, that is, reduces the data arraysizes by excluding zero elements (step S2). The reduction of the dataarray sizes will be described with reference to FIG. 4. As is clear fromthe Bayer pattern image illustrated in FIG. 3, if the gaps within theimages are closed for the R and B channels, the arranged numbers of datawill become H/2 in the vertical direction and W/2 in the horizontaldirection. Therefore, the reduced data arrays for the R and B channelsbecome those indicated by (1) and (4) in FIG. 4. That is, the reduceddata arrays of the R and B channels will have ¼ the number of data ofthe original image.

Meanwhile, data of the G channel is arranged in the state indicated by(2) of FIG. 4. After the gaps in the image are closed, the data array isrotated 45 degrees. Thereby, the reduced data array indicated by (3) ofFIG. 4 is obtained. The number of data in this reduced data array isH/√2 in one direction, and W/√2 in the direction perpendicular thereto.That is, for the G channel, the number of data in the reduced data arraybecomes ½ that of the original image.

Next, the image reconstruction processing section 15 administers imagereconstruction filtering processes using deconvolution filters 16 havingfilter sizes corresponding to the size of each array, onto the reduceddata arrays obtained in the manner described above for each of the R, G,and B channels (step S3). The deconvolution filters 16 have propertiesinverse to the blur properties of the lens optical system 12, that is,the point spread function thereof. By performing the filteringprocessing using the deconvolution filter 16 having such properties,image data that bear images in a focused state, in which blur has beenresolved, are obtained. Note that this type of image reconstructionfilter is described in detail in U.S. Patent Application Publication No.20090147111 and Japanese Unexamined Patent Publication No. 2009-089082.Such a known image reconstruction filter may be employed in the presentinvention.

(1), (2), and (3) of FIG. 5 each denote the filter size (toward the leftof the drawing) determined for the R, G, and B channels, respectively,and the diameters of spread of the deconvolution filters 16 (toward theright of the drawing). Note that the diameter of spread of thedeconvolution filters 16 are φ/2, φ/√2, and φ/2 for the R, G, and Bchannels, respectively.

Next, the image reconstruction processing section 15 performs a processto return the data, which have undergone the filtering process, to theBayer pattern (step S4). Note that with respect to the G channel, datawhich have undergone the filtering process are rotated −45 degrees toreturn the orientation of the data array to the original orientation,then returned to the Bayer pattern.

Data which have been returned to the Bayer pattern in this manner aredenoted by (1) of FIG. 6. The data, which have been returned to theBayer pattern, are sent to the interpolation processing section 18 ofFIG. 1. The interpolation processing section 18 administers aninterpolation process onto the data in the Bayer pattern (step S5). Thisprocess is generally referred to as a synchronizing process, and is aprocess that interpolates spatial shifts among color signals that occurdue to color filter arrangement, to calculate color data for each point.Thereby, a three color image signal that represents data for each ofthree colors R, G, and B is obtained, as denoted by (2) of FIG. 6.

As described above, in the present embodiment, the image reconstructionprocessing section 15 collects data while excluding zero elements fromeach of the R, G, and B channels. Thereby, reduced data arrays, in whichthe amounts of data for the R and B channels is ¼, and the amount ofdata for the G channel is ½, are generated. The filtering processesusing the deconvolution filters 16 are performed onto data thatconstitutes the reduced data arrays for each of the R, G, and Bchannels. Therefore, zero elements of the image reconstruction filterare eliminated from the calculation processes, which reduces thecalculation costs. Further, the configurations of calculating programsand circuits can be simplified.

In addition, the imaging device of the present invention performs theimage reconstruction process onto the G channel, which is a combinationof the Gr channel and the Gb channel. Therefore, loss of high frequencycomponents which occurs when the Gr channel and the Gb channel areprocessed separately can be prevented, and a more highly detailedultimate image can be obtained.

Further, in the present embodiment, data of the G channel are rotatedand rearranged. Therefore, the configurations of the calculatingprograms and circuits can be further simplified, reducing system costsand calculating costs even more.

Next, a second embodiment of the present invention will be describedwith reference to FIG. 7. In the second embodiment, the reduced dataarrays for the R and B channels denoted by (1) and (3) of FIG. 7 are thesame as those illustrated in FIG. 5. However, the reduced data array ofthe G channel is of an arrangement different from that of FIG. 5. Thatis, in the second embodiment, data of the G channel are collected whileexcluding zero elements in the horizontal direction (aligned toward theleft in FIG. 7) to generate the reduced data array. In this case, thenumber of data in the array will become H in the vertical direction andW/2 in the horizontal direction. Accordingly, the number of data is ½that of the original image.

Note that (1), (2), and (3) of FIG. 7 also denote the diameter of spreadof deconvolution filters for each of the R, G, and B channels,respectively toward the right side of the drawing. In this case, thediameters of spread of the deconvolution filters for the R and B channelare both φ/2. The spread becomes elliptical for the G channel, and thelength of the major axis is φ, while the length of the minor axis isφ/2.

Here, the deconvolution process to be administered onto the G channel,which is a combination of the Gr and Gb channels will be that as denotedby (1) of FIG. 8 if considered in the original Bayer pattern state.However, the data of the G channel is rearranged into the reduced dataarray such that the vertical size of is maintained, while the horizontalsize is reduced to ½, as illustrated in FIG. 7. Therefore, the positionsof corresponding elements of the deconvolution filter will be shifted.This problem can be resolved easily, as will be described hereinbelow.That is, the arrangement of elements of the deconvolution filter is alsorearranged, to obtain a filter having a Gr cell at the center as denotedby (2) of FIG. 8, and a filter having a Gb cell at the center as denotedby (3) of FIG. 8. Further, the filter which is employed is switched forcases in which the central element of the convolution process is a Grelement or a Gb element. With respect to the data arrangement followingthe filtering process, the data may be arranged according to the filterwhich was utilized, as denoted by (4) of FIG. 8. Note that (2) and (3)of FIG. 8 respectively correspond to the region denoted by the boldsolid line and the region denoted by the bold broken line in (1) of FIG.8.

Next, a third embodiment of the present invention will be described withreference to FIG. 9. In the third embodiment, a reduced data array isgenerated for the G channel by excluding zero elements while aligningdata toward the left, as denoted by (1) of FIG. 9. Then, deconvolutionfilters having sizes indicated by the bold solid line and by the boldbroken line of FIG. 9 are employed to perform image reconstructionprocesses on data that constitute the reduced data array. That is, inthe third embodiment, the elements of the deconvolution filters arearranged such that the Gr rows and Gb rows are shifted by one element,in order to absorb positional differences among Gr data and Gb data inthe horizontal direction during image reconstruction processes performedon the G channel, which is the Gr and Gb channels combined.

These deconvolution filters are denoted by (2) and (3) of FIG. 9. Theformer is that in which a Gr cell is positioned at the center, and thelatter is that in which a Gb cell is positioned at the center. In both(2) and (3), the blacked out portions indicate zero elements.

Note that the direction in which the Gr row and the Gb row are shiftedis reversed depending on whether the data of the central pixel duringthe convolution calculation is Gr or Gb. For this reason, the two typesof deconvolution filters described above for cases in which the data ofthe central pixel is Gr and for cases in which the data of the centralpixel is Gb. The two types of filters are selectively utilized for theaforementioned two cases during the convolution process. Data obtainedby the image reconstruction process using these deconvolution filtersmay be arranged as illustrated in (4).

Note that correction gain, which are unique values for the Gr cells andthe Gb cells, may be overlapped onto each element of the imagereconstruction filters denoted by (2) and (3) of FIG. 8, for example. Ifthese image reconstruction filters are used, sensitivity differencesamong the Gr cells and the Gb cells can be automatically corrected. Byadopting this configuration, a separate process that corrects thesesensitivity differences is obviated, which is preferable from theviewpoint of expediting image processes.

What is claimed is:
 1. An imaging device, comprising: a lens opticalsystem, for focusing light from a subject; color imaging means equippedwith a Bayer pattern color filter, for imaging an image of the subjectfocused by the lens optical system; and an image processing section, forperforming a filtering process in which data output by the color imagingmeans is passed through an image reconstructing filter having propertiesinverse blur properties of the lens optical system, then performing asynchronization process; wherein the image processing section collectingdata excludes zero elements for each of R, G, and B channels, togenerate reduced data arrays in which the amount of data is ¼ for the Rand B channels, and the amount of data is ½ for the G channel, andadministering the filtering process using the image reconstructionfilter onto data of the reduced data array for each of the R, G, and Bchannels; wherein the image processing section is configured to: collectdata excluding zero elements, and rotates the data array 45 degrees togenerate the reduced data array for the G channel; administer thefiltering process employing an image reconstruction filter, of whichdata have similarly been rotated 45 degrees, onto data that constitutesthe reduced data array for the G channel; and rotate the data obtainedby the filtering process 45 degrees, to return the orientation of thedata array.
 2. The imaging device as defined in claim 1, wherein:correction gain, for correcting differences in sensitivities among Grcells and Gb cells, are overlapped onto each element of the imagereconstruction filter which is employed with respect to data of the Gchannel.